Software applications and products are typically subjected to various test cases prior to and in the early stages of development and deployment. For example, during bug fixing or feature enhancement, a fairly typical design practice involves running (or re-running) all of the test cases in a test plan for the software application to help validate (or re-validate) the application and make sure the new code changes do not “break” any existing functionality. The software code for an application is generally stored in a code repository. When a change is made to a portion of the software code, it is “checked-in” or “committed” to the repository. Upon each committal (“commit”), the test plan may execute (or run) one or more test cases. It may be unnecessary to re-run all the test cases in the test plan because a given commit need not necessarily affect how all test cases perform. For example, some commits have limited scope and, as a result, only a subset of test cases need to be run (or re-run) to ensure that the commit did not “break” any existing functionality.
The number of test cases that are executed for a software application depends on several factors, such as how large and/or complicated the program is, whether/how it relates to mission-critical or other key operations, the extent to which it interacts with other systems, etc. An enterprise product is typically subjected to thousands of test cases prior to deployment. Moreover, depending on the application being tested, the platform for running the test cases may vary. For example, a change to the code of software application that can be deployed on several different devices (e.g., smartphones, tablets, laptops, etc.) may require running the test cases associated with the code change(s) (commits) on each device that supports the software application. As a result, testing of large and/or complicated software application on different devices can be challenging because it can require execution of numerous test cases that take a long time to execute on a limited number of available devices. The problem gets compounded due to continuous development where multiple software developers are checking-in code (i.e., multiple commits) at the same time. Also, certain commits may result in regression where a software bug may make an application feature to stop functioning as intended. Regression testing may be required to prevent regression.
Examples of current test methodology for testing a software application that can be deployed on several different device involve waiting for a certain number of commits (e.g., 20) of new code before running test cases. This results in peaks and valleys in demand for devices on which the software application is being tested because developers can check in code at random times. This may also result in bad code to get deployed to a production environment if the code change is not tested in a timely and efficient manner. Thus, there exists a need for a software testing system and method that overcomes these and other drawbacks of existing test methodologies. A new system and method is disclosed that dynamically schedules testing based on when the test cases were last executed. Test cases are ordered based on the which commit they were last executed on (e.g., test cases with older untested commits may be scheduled first). The order is then adjusted based on one or more of the following factors: a priority multiplication factor, a maximum amount of time since last execution, and a maximum number of commits since last execution.
The techniques introduced here may be better understood by referring to the following Detailed Description in conjunction with the accompanying drawings, in which like reference numerals indicate identical or functionally similar elements.